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Abstract. Let G/H be a hyperbolic space over R, C or H, and let X be a maximal compact 
subgroup of G. Let D denote a certain explicit invariant differential operator, such that the non- 
cuspidal discrete series belong to the kernel of D. For any L^-Schwartz function / on G/H, we 
prove that the Abel transform A{Df) of Df is a Schwartz function. This is an extension of a 
result established in [2] for ii'-finite and K n //-invariant functions. 



1. Introduction 

The Radon transform R on the hyperbolic spaces G/H, 



Rf^f fi-nH)dn, 

JN* 



IN 

where N* C G is a certain unipotent subgroup, and the associated Abel transform A, were introduced 
and studied in [T] and [5]. Generahzing Harish-Chandra's notion of cusp forms for real semisimple 
Lie groups, a discrete series is said to be cuspidal if it is annihilated by the Radon transform. In 
contrast with the Lie group case, however, non-cuspidal discrete series exist. For the projective 
hyperbolic spaces, these are precisely the spherical discrete series, but for some real non-pro jective 
hyperbolic spaces, there also exist non-spherical non-cuspidal discrete series. 

Let C^{G/H) denote the space of L^-Schwartz functions on G/H. Except for some boundary cases, 
A maps C^{G/H) into Schwartz functions in the absense of non-cuspidal discrete series. On the other 
hand, Af can be explicitly calculated for functions / belonging to the non-cuspidal discrete series. 
To complete the picture, we prove below that A essentially maps the orthocomplement in C^{G/H) of 
the non-cuspidal discrete series into Schwartz functions. To be more precise, let Ap = A-|-p^, where 
A denotes the Laplace-Beltrami operator on G/H , and consider the G- invariant differential operator 
D = Ap{Ap — Xf) . . . {Ap — A^), where Ai, . . . , Ar are the parameters of the non-cuspidal discrete 
series. Then A{Df) is a Schwartz function. This extends our previous result, [U Theorem 6.1 (vi)], 
valid only for the dense G-invariant subspace of C^{G/H) generated by the if-irreducible {K n H)- 
invariant functions, to all Schwartz functions. 

In [2] we also considered the exceptional case corresponding to the Cayley numbers O. We expect 
our new result to hold for this case as well, but we have not been through the rather cumbersome 
details. 

The second author wants to thank Professor Vladimir Molchanov for the invitation to visit Tam- 
bov University, where our results were first reported, in November 2012. We would also like to thank 
Hcnrik SchlichtkruU and Job Kuit for helpful discussions and comments. 



2. The Radon transform 

In this section, we define the Radon transform and the Abel transform for the projective hyperbolic 
spaces over the classical fields F = M, C and H. We have tried to keep the presentation and notation 
to a minimum, see [1] and [2] for further details (including results and proofs). 
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Let X t-^ X he the standard (anti-) involution of F. Let p > 0, g > 1 be two integers, and consider 
the Hermitian form [•, •] on Fp+'+^ given by 

[x, y] = xiy^ + ■■■ + Xp+iljp^^ - Xp+2yp+2 a^p+i+g+i!/p+i+g+i, {x, y E FP+«+2). 

Let G = \J{p+l,q + l;¥) denote the group {p + q + 2) x {p + q + 2) matrices over F preserving [■,■]. 
Thus U(p+1,(7 + 1;M) = 0{p+l,q+l), V{p+l,q + l:C) = U(p+l,g+l) and U(39 + 1, g + 1; H) = 
Sp(p+l,(j+l) in standard notation. Put U(p;F) = U(p,0;F), and let K = \J{p+l;¥) x U(g+1;F) 
be the maximal compact subgroup of G fixed by the Cartan involution on G. 

Let .To = (0, . . . , 0, 1)-^, where superscript T indicates transpose. Let H = \J{p + 1, q; F) x U(l; F) 
be the subgroup of G stabilizing the line F • xq in Fp+''+^. The reductive symmetric space G/H can 
be identified with the projective hyperbohc space X = X{p + 1, g + 1; F), 

X= {z GFf+«+2 ; [z,z] = 

where ^ is the equivalence relation z ^ zu, u G F*. 

Let Xt, for t £ M, denote the element in the Lie algebra g of G with value t in the (l,p + q + 2)'th 
and {p + q + 2, l)'th matrix entries (the two opposite corners in the anti-diagonal), and otherwise. 
Let Oq denote the Abelian subalgebra given by {Xt | i S R}, let a* = exp(Xt) denote the exponential 
of Xt, and also define = exp(aq). 

Let (considered as row vectors) 

u = {ui, . . . , Up) e F^ and v = {vq, . . . ,vi) e F'', 
and let w e LnF (i.e., w = for F = M). Define Nu,v,w G as the matrix given by 
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Then exp{Nu,v,w) ^ I + Nu,v,w + ^/'^N^,v,nn ^^'^ ^ small calculation yields that 

(1) a(exp(7V„,„^.„)-a;o = (sinh < + l/2e*(|Mp - |wp) + e*u;, u^; -u^, cosht + l/2e*(|u|^ - |u|^) + e*u;)'^, 
for any t eR. 

Define the nilpotent subalgebra n* as follows, for p > q, 

(2) n* = {iV„ : u = {-W, u'), v e F«, u' G F^-^}, 
and, for p < q, 

(3) n* = {iV„,„,„, : V = {-W, v'), u e F^ , v' £ F«-f }, 

where vT , means that the order of the indices is reversed. By abuse of notation, we leave out the 
superscript in what follows. 

We finally also define the following p-factors. Let d = dimg F, and let pq = ^{dp + dq + 2{d— 1)) € 
E, and pi = ^{{\dp - dq\ + 2{d - 1)) g E. 

Let N* = cxp(n*) denote the nilpotent subgroup generated by n*. For functions / on G/H, we 
define, assuming convergence, 

(4) Rf{g)^ I f{9n*H)dn* (geG). 

Let / G C'^{G/H), the space of L^-Schwartz functions on G/H. From [1] and P], we know that 
the Radon transform Rf is a smooth function. Also, the integral defining R converges uniformly on 
compact sets, and R is G- and g-equivariant. 

We define the associated Abel transform A by Af{a) = a^^Rf{a), for a G Aq. We are mainly 
interested in the values of Rf and Af on the elements a^, and thus define Rf{s) = Rf{as), and. 
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similarly, Af{s) = Af{as), for s e M. Let A denote the Laplace-Beltrami operator on G/H. Then, 
for /GC2(G/iJ), 

(5) A{Af ) ^ - Af (.GM). 

Finally, for i? > 0, let C'^{G/H) denote the subspace of smooth functions on G/H with support 
in the (ii'-invariant) 'ball' {kus ■ xq \ \s\ < R} of radius R. Similarly, let C^(M) denote the subspace 
of smooth functions on R with support in [—R, R], and let <S(R) denote the Schwartz space on K. 

3. The discrete series and the Abel transform 

Let g > 1, or d > 1. The discrete series for the projective hyperbolic spaces can then be 
parametrized as 

{Ta I A ^{dq - dp) - 1 + MA > 0, e 2Z}, 

see [T] and [2]. The spherical discrete series are given by the parameters A for which px < 0, including 
the 'exceptional' discrete series corresponding to A > for which p\ < 0. 

For q = d = 1, the discrete series is parameterized by A G IR\{0} such that |A| + pq G 2Z, and 
there are no spherical discrete series. 

The parameters A are, via the formula A/ = (A^ — p^)f, related to the eigenvalues of A acting 
on functions / in the corresponding representation space in L'^{G/H). 

We have a complete classification of the cuspidal and non-cuspidal discrete series for the pro- 
jective hyperbolic spaces, also including information about the asymptotics of the Radon and Abel 
transforms: 

Theorem 1. Let G/H be a projective hyperbolic space over R, C, H, with p > 0, q > 1. 

(i) If d{q — p) <2, then all discrete series are cuspidal. 

(ii) If d{q — p) > 2, then non-cuspidal discrete series exists, given by the parameters A > with 
px < 0. More precisely, if ^ f G C^iG/H) belongs to Tx, then Af{s) = with C ^0. 

(iii) Tx is non-cuspidal if and only if Tx is spherical. 

(iv) Ifp > q, and f G C^{G/H), for R>0, then Af G C^(R). 

(v) Ifd{q~p) < 1, and f G C^{G/H), then Af G <S(R). 

(vi) Assume d(q—p) > 1. Let D be the G-invariant differential operator Ap(Ap — A^) . . . (Ap — A^,), 
where Ai, . . . , A^ are the parameters of the non-cuspidal discrete series, and Ap = A + p^. 
Then A{Df) G 5(R), for f G C^{G/H). 

The above theorem is almost identical to [2l Theorem 6.1], except for item (vi), which was 
only proved for functions in the (dense) G-invariant subspace V of C^{G/H) generated by the K- 
irreducible {K n _ff )-invariant functions. Additionally, O Theorem 6.1] furthermore included the 
exceptional case corresponding to the Cayley numbers O. 

Theorem [T] (including the reformulation of (vi)) also holds for the real non-pro jective spaces 
SO{p + l,q + l)e/SO{p -\- l,q)e, except for item (iii), due to the existence of non-cuspidal non- 
spherical discrete series corresponding to negative and odd values of px in the exceptional series, see 
[II Section 5]. 

The conditions in (vi) essentially state that Af is a Schwartz function if / is perpendicular to all 
non-cuspidal discrete series. The factor Ap, however, seems to be necessary (except in the real case 
with q ~ p odd), even for the case d{q — p) = 2, where there are no non-cuspidal discrete series. 

In the next section, we prove Theorem [Ijvi) . 

4. Proof of Theorem mvi) 

First we note, following [2j Section 10], that the Schwartz decay conditions are satisfied near — oo 
for A{f), and thus also for A{Df ). This leaves us to study the Abel transform near +oo. 
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Let / e C'^{G/H), and write f[x] = f{gH), where x = g ■ xq- From JT]) and (O, we get 
Rfis) = / fiasn*H)dn* = 



I 

Jr" 



/[(sinhs — l/2eJ\v'\'^ + e'''w,u; — m, —v',coshs ~ l/2e/\v'\'^ + e^if)] dv' dudw. 
Let u' = \v'\v, V = — sinhs + l/2e''|u'|^, such that = 1 + 2e~^u — e~^*, and lu ^ e'^w. Then, 
Rf{s) = e-'^' f f f[{w - V, u; ~u, -(1 + 2e-'v - e'^'y/^v, e'' -v + w)] 

X (1 + 26-^1. - e-2s)(rf9-dp)/2-i ^„ dvdudw, 

where S*" is the unit sphere in W. 

We win use the identification of X = X(p + l,q+ 1; F) with 

X=={zeFP+«+2 : [z,z] <0}/-, 

and identify a function / on X with a homogeneous function on {z G Fp+''+^ : [z, z] < 0} of degree 
zero. 

We now identify Fp+'?+2 with 1^^(^+9+2) such that the coordinates satisfy Rezj = Xdj, for j = 
1, . . . ,p + q + 2. Consider the real hyperbohc space X = {z G : [z, z] = —1}. The group 

G = 0{d{p + l),d{q + 1)) acts transitively on X. Let^A' = 0{d{p + 1)) x 0{d{q + 1)) denote 
the standard maximal compact subgroup of G. Let U{t), respectively C/(6), denote the universal 
enveloping algebra of the Lie algebra t of K, respectively of the Lie algebra t of K. 

Lemma 2. Let U G U{1), then U maps C^{G/H) into itself. 

Proof. The lemma is obvious for d ^ 1. So assume d > 1. We note that any element x G X can be 
written as a; = fca • xq, where k G K, and a = a^, s > 0. Let H = 0{d{p + 1), d{q + 1) — 1), and let 
m denote the commutator of in the Lie algebra of K D H. Then 4 = 6 + m. 

Let Uk = Ad{k)U, for fc G if, then Uf = {Ad{k-^)Uk)f . By the Campbell-Baker-Hausdorff 
formula, there exists an element [/° G U{t), such that Uk = modulo the left ideal generated by 
m. This implies that Uf[ka ■ xq] = {Ad{k~^)U^)f[ka ■ xq]. The map k i-^ Ad{k~^)U^ is continuous 
into a finite dimensional subspace of U{t), and we can write Uf[ka ■ xq] = {Ad{k~^)U^)f[ka ■ xq] = 
Y,iUi{k)Uif[ka-xo], for a finite set of elements Ui G U{i) and continuous coefficients Ui{k). It follows 
that Uf is in C^{G/H). □ 

Define for t = (ti,t2,i3) G IR"^, the auxiliary function 



Gf{ti,t2,t3) = / f[{w + ti,u;~u,t2V,t3 + w)]dvdudw, 

Jgdg-dp-l xRdpxRti-l 

and, with the identification z ~ e^'*, define the function F{z) = e/''Rf{s). Then, since sinhs = 

— (z — z^-'-)/2, we get 

/•oo 

(6) F{z)= / Gf{~v,~{l + 2zv - zY^^,z ^v)(l + 2zv - z^)^'^'i-'^P'>/^-Uv. 

Lemma 3. The function G f is homogeneous of degree dp + d — 1, for t E {t \ t^ — t2 — t^ < 0}, Gf 
is even in t2, and satisfies G f{~ti,t2, —is) = Gf(ti,t2, is). 

Let X be the differential operator on given by t3d/dt2 — t2d/dt3. For all f G C^(G/ H), and 
all k,N £N, there exists a constant C, such that 

\X''Gf{t)\ < G{tl + t2)-rf(9-p)/4(i + iog(t2 + tl^yN^ 
for all t e {t\tl-tl-tl = -1}. 



ASYMPTOTICS FOR THE RADON TRANSFORM 



5 



Proof. The first statement follows from the homogeniety of / and the definition of G/. 

As before we identify ¥p+i+^ with Rd{p+q+2) ^ Define, for i = d{l + 2p) + 1, . . . ,d{l + p + q), the 
differential operator Dif[x] = a;rf(p+5+2)9/9a;i/[x] — Xid/dxd(p+q+2)f[x]- This operator is defined by 
the left action of an element Ti in 0{d{q + 1)) (with value 1 in the last entry of the i'th row, value 
— 1 in the last entry of the i'th coloum, and otherwise), and Lemma [2] thus gives that Di maps 
C^{G/H) into itself. 

Let now V = {vd(i+2p)+i, ■ • ■ , Vd(i+p+q-)) e S'^^i-P^-^. The operator 

-I'll — '^i=2+2p''i^" 

also maps C^{G/H) into itself, and 

l>V/N| <rf(g-p)max(|A/NI). 

i 

Applying the operator X to the integrand in the definition of G/, we get 

= Y-f[ti,u; -u,t2V,h]. 

The inequality for f follows from repeated use of this formula and from the asymptotic estimates 
of functions in C^iG/H). □ 

In particular, it follows that the function v i-^ X'^G f{—v, —1, —v) has the same parity as k. 

Lemma 4. Let fco be the largest integer such that fco < (dq — dp)/2, and let e ~ {dq — dp)/2 — kg. 
Define t^t{z,v) = {-v,-{l + 2zv ~ z'^f/'^ , z - v). Then 

(i) For k < ko, the function v ^ d'' /dz''{G f{t{z,v))\{l + 2zv - 22)|(dg-dp)/2-i) uniformly 

integrable over K for z < \. 
(u) For k < ko odd, the function v ^ d'' /dz''{G f{t{z,v)){l + 2zv - ^2)(dg-dp)/2-i) 
function of v for 2 = 0. 

Proof. Notice that tl~t^~tl = —1 and t^+tl = 1 + u^, iov t ~ t{z,v), and that the integral © is 
uniformly convergent for < z < A' < oo. The same holds with G/ replaced by X^Gf. 

Repeated use of the formula 9/az(G/(<(z, ?;))(l+2zt>-z2)") = -XG f{t{z,v)){l+2zv-z^)"'-^''^ + 
2aG f{t{z,v)){\ + 2zv — — v) yields (i), and together with the parity properties of X'^Gf 

also gives (ii). □ 

We notice that e = 1 if d{q — p) is even, and e = 1/2 if d{q — p) is odd, i.e., if d = 1 and q — p is 
odd. 

For fc < fco, the derivatives /dz^ of G/(t(z, w))(l+2zw— are zero atu = -sinhs = 
i(z — z~^), whence the integrand is at least fco times differentiable near z = 0, and we can compute 
the derivatives d*' /dz^F{z) by differentiating under the integral sign in (|6|). 

If fc'o > 0, we can use Taylors formula to express F{z) as a polynomial of degree fco — 1, plus a 
remainder term involving d^° / dz^° F {£) , for some < ^(z) < z, 

F{z) = Co + ciz + C2z2 + ... + Cfe„_iz^«-i + i?fc„(e)z^°, 

where < ^ < z, and 

1 di 

(G/(t(z,w))(l + 2zi;-z2)(^9-'*P)/2-i)dw, 

2=0 



j! y_oo dz3 



for i G {0, . . . , fc'o — 1}. The remainder term is given by: 



1 f°° 

fco! 7(5-^-1 



)/2 dzk« 



{Gf{t{z, V)){1 + 2ZV - 22)(<i9-dp)/2-l^ 
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Consider Af{s) = eP^^Rf(s) = z^^p^''^'^ F{z), which is equal to 

Here we have used that pi — d ~ d{q — p)/2 — 1. For j even, the exponents —d{q — p)/2 — 1 — j, for 
j G {0, . . . , fco — 1}, correspond to the parameters Ai, . . . , for the non-cuspidal discrete series, and 
Cj = for j odd, since the integrand is an odd function. 

For the real non-projectivc hyperbolic spaces the condition concerning the parity j docs not hold, 
but in that case aHthe exponents —d{q—p)/2 — l—j, for j G {0, . . . , fco — 1}, correspond to parameters 
Ai, . . . , Ar for the non-cuspidal discrete series, see [1] Section 3] 

From the definition of the differential operator D and ([5]), we see that A{Df) at most has a 
contribution from the remainder term, and further that A{Df) does not have a constant term at oo, 
due to the term d"^ /ds^. If e = 1/2, the remainder term e'^/^'^i?;;^ (^(s)) is clearly rapidly decreasing, 
and we are thus left to consider the case e = 1, in which case fcg = d{q — p)/2 — 1. 

Consider the constant term Cr^^^ = \va\s^oo Rka{^~^), which could be non-zero. We want to 
show that RkoiO — ^Rkg is rapidly decreasing at +oo, where ^ — ^(s), with < ^ < e"''. We also 
include the case fcp = 0, where we put ^ = e~*. 

Define H{z,v) = ^{G f{t{z,v)){l + 2zv - z^f°). Then, for ^ < 2 < 1, 

/•oo /•(?-«"')/2 

Rko{0-CR,^= {H{^,v)-H{0,v))dv+ H{0,v)dv ^ hiO + hiO- 

For Ii{£,), there exists ^1 = £,i{£,,v) < ^, such that H{£^,v) — H{Q,v) ~ ^d/dz\^^^^H{z,v), and we 
get: 

d_ 

dz 

By LemmaHl the integrand is uniformly integrable for z < 1, and we conclude that /i(^) is bounded 
by Ce-". 

For s large, the function -ff (0, v) is for every G N bounded by 

|if(0, V)\ < C{1 + v2)-(-d(9-p)/4|„|feo log(l + „2)-JV^ 

for some positive constant C . Using this, we find that 

/•oo 

h{z) <C v-\log{v))-^ dv = C{N - l)-i(log(sinh.s))-^+i < Cs-^+\ 

J sinh s 

It follows that RkoiO ~ ^Rko is rapidly decreasing at +00, whence A{Df) is rapidly decreasing at 
-l-oo, which finishes the proof of Theorem [TJ 
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